Algorithms for Identification and Categorization 
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This talk reports on a series of efforts during the 
last decade aimed at modeling in a computer the 
processing of patterns in the brain. In particular, 
a main recent interest is to design fast and reliable 
algorithms for "restoring" a pattern, namely, af- 
ter degrading highly a pattern, to identify it. The 
challenge is to go beyond familiar methods, includ- 
ing Hopfield-like neural networks J^ 2, which, in 
spite of a great theoretical interest, are hampered 
in practice by the occurrence of mixture states, 
slow processing, limited capacity, and inadmissible 
statistical errors. Furthermore, we are concerned 
with an extended definition of "pattern" , namely, 
we plan to deal from a high-resolution color pic- 
ture to a complex pattern of behavior or the large 
set of qualities which serve to accurately identify 
a complex company, for example. This is relevant, 
in particular, to sociology, the focus in this meet- 
ing. However, the discussion below will avoid any 
specific application — except, eventually, for illus- 
trative purposes. 

Compared to machines, the hunran brain per- 
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forms impressively regarding to our recent inter- 
est. Unfortunately, the brain strategy in these 
processes is not yet well understood. Neverthe- 
less, there are now some indications which we shall 
take here as the basic hypothesis to build a model. 
Firstly, that processing of patterns is the result of 
cooperation between many units. Next, that as- 
sociative memory in neurobiological systems is the 
consequence of, say, an essentially stochastic dy- 
namics. That is, taking along the description to 
well-defined grounds, dynamics induces stochastic 
jumps between pure attractors, i.e., fixed points as 
the ones in the Hopfield model. Even more, the 
jumping is often restricted to the elements in one 
or a few of the classes in which one may assume 
the whole set of attractors to be partitioned. This 
possibility is interesting because it implies catego- 
rization, namely, classification of the stored pat- 
terns according to some criteria — e.g., the degree 
of correlation between the elements in the class(es). 
Such behavior has already been reported in sev- 
eral contexts, including the monkeys cortical activ- 
ity and the insects early olfactory processing |3|- 
[5. In fact, there is some evidence that recognition 
involves a previous process of class discrimination 
(indeed one may recognize a person as familiar and 
not being able to recall its identity) 8 . 

The origin for the essential stochasticity in ac- 
tual systems is likely to be at the synapses, which in 
practice determine much of the complex processing 



of information in the brain |5]. That is, the neural 
activity, which is intrinsically stochastic, further re- 
quires the continuous competition of (fast) "synap- 
tic noise" , which in turn is activity dependent, to 
accomplish the efficient transmission of informa- 
tion and a variety of computations ^1 E] . Conse- 
quently, pursuing recent efforts ^21 j we found sen- 
sible investigating the effect of stochastic synapses 
on the fixed points of the retrieval processes in 
appropriate attractor neural networks. With this 
aim, we believed it convenient to model the synap- 
tic noise based on the observation that synapses 
endure short-time depression. That is, periods 
of elevated presynaptic activity decrease the neu- 
rotransmitter release and, consequently, the post- 
synaptic response is depressed, which is likely to 
positively influence the transmission of information 
0,1111-1111. 

We shall briefly illustrate the above ideas in 
a simple model system which generalizes sev- 
eral proposals. Consider a set of units (neu- 
rons, social agents, etc.) that change ran- 
domly of state with time. Their stochastic- 
ity is regulated by a "temperature" parameter, 
Tq. Each unit is connected to each other with 
links, synapses, etc. whose intensities or weights 
change with time, also at random but regulated 
by a different "temperature" , Ti . This sort of 
competition was flrst considered in Refs.^B] - 
|21|. To be specific, let N binary (say) neurons 
with configurations S = {s^ = ±1; i = 1, . . . , N} 
connected by (say) synapses of weight Wij e 
SR; i,j — 1,...,N. The probability of the 
state O = (S,W), with W ={wij}, at 
time t satisfies the familiar master equation, 

i.e., dtP{n) = j:n'[c{^'^n)Ptin') - 

c (O ^ O') Pt (fl)] , with a transition rate which 
describes the indicated competition, namely 
c(f2->f2') = p c^ (S -> S') (5 (W - W) + 
(1 - p) c^ (W -> W) S{S~- S') . The elementary 
rates c^ and c^ are typically chosen just by 
looking for simplicity ^J. For instance, one 
may take the former as a factorizable function of 
2 (s./To) h^iS), where hf'iS) = E,_^, w.jSj is the 
net (pre)synaptic current arriving to — or local field 
acting on — the (postsynaptic) neuron i. Here, Wij 
is determined in a previous learning process, e.g., 
by the Hebb's rule 22 from M patterns which in 
this way remain stored in the system. It is to be re- 
marked that, due to the generality we call for here. 
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Figure 1: Parameter space for 16384 neurons and 
3 stored, highly-correlated patterns showing P and F 
phases, as defined in the text, and regions in which the 
state keeps jumping between patterns. The jumps are 
either uncorrelated or time-correlated for regions 0(I) 
and 0(11), respectively. 

these patterns should be as diverse as to include 
from a set of orthogonal or near orthogonal pat- 
terns to a set of patterns that are highly correlated 
to each other. 

A particularly interesting case occurs for p — > 0. 
In fact, this mimics rapid fluctuations in accor- 
dance with the observation of fast noise mentioned 
above. Furthermore, the model can then be solved 
analytically for some choices of parameters. One 
is lead in this limit, after rescaling time tp -^ t, 
to a situation in which the neurons evolve as in 
the presence of a steady distribution for the synap- 
tic fluctuations. The transition between states is 
then governed by an effective rate that involves 
the noise (effective) distribution. A principal re- 
sult is that the ensuing stationary states are out of 
equilibrium |21|. This is because of the competi- 
tion between the neural activity and the synaptic 
noise, which impedes reaching asymptotically equi- 
librium. This was flrst studied in some detail in 
Refs. |17lll8| . and analysis of the intriguing behav- 
ior this may show concerning recognition was first 
reported in I^efs. pil^ . 

Also remarkable is that, under appropriate con- 
ditions, which include parallel updating — as it is 
usual in cellular automata — , this system exhibits 
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Figure 2: This shows, as a function of time, the order /i 
of the pattern which more closely resembles the current 
state of a network with parameters in region 0{II). 
This is for M = 9 correlated patterns stored in a set of 
10* fully connected neurons with To = 0.05 and Ti = 
1.25 times the corresponding critical values. This shows 
how the neural automata constantly jumps between the 
stored patterns. The jumping is apparently random, 
but the system detects a family — consisting of patterns 
7, 8 and 9 whose elements are more correlated to each 
other than with the rest. 



various retrieval phases. This is illustrated in fig- 
ure n In addition to a disordered phase (P) and 
a phase with associative memory (F), there is a 
region in which one observes a continuous stochas- 
tic jumping between attractors (O). Depending on 
the parameter values, this jumping may be rather 
complex 1231; see figure|21 On the other hand, there 
is no region in the parameter space with mixture 
states such as the ones that hamper recognition in 
the Hopfield case 2^, J19_ . Also noticeable is that 
phase transitions may be of first-order, which im- 
plies that the recognition process may occur with 
negligible error |2U| . 

The same model was recently studied to show 
that the network topology importantly matters. It 
is unlikely that natural evolution leads to fully con- 
nected networks, and it was shown j24| that, in gen- 
eral, the capacity to store and retrieve is higher for 
a scale-free topology than for a comparable highly 
random-diluted network. 

One may explicitly show that some of the jump- 
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Figure 3: Time evolution of the overlap, as defined 
in the main text, between the current state and the 
stored pattern in Monte Carlo simulations with 3600 
neurons at T = 0.1. Each graph shows different curves 
corresponding to evolutions starting with different ini- 
tial states. The two graphs are for 5 = 0.3 and "1? = 1 
(right) and $ = — 1 (left), the former corresponding to 
the Hopfield case lacking the fast noise. This shows the 
important effect noise has on the network sensitivity 
to external stimuli. The same behavior occurs as one 
varies the external stimulation 5 . 



ing phenomena in the model — e.g., the one shown 
in figure |21 — is close to the reported "familiar- 
ity discrimination" S^ by tuning the model de- 
tails. For example, one can assume that the synap- 
tic intensities are now Wij — WijXj, where Wij is 
fixed in a previous learning, as before, and Xj is 
a stochastic variable. Let us assume that, in the 
limit p — > 0, the steady noise effective distribution 
is a product of functions P{xj\S) — C{^) ^{^j + 



$) + [l-C(m)] S{xj - 1). Here, rii = m(S) is 
the M— dimensional vector of overlaps m'^(S) = 
^E.s^er, where ^ = {Cf = ±1, * = 1, • . • , iV} 
are the M stored patterns, and ( (m) stands for a 
function of rii to be determined. This choice aims 
at modeling the reported short-term synaptic de- 
pression ^1^]. That is, increasing the mean fir- 
ing rate results in decreasing the synaptic weight, 
because the depression effect here depends on the 
overlap vector which measures the net current ar- 
riving to postsynaptic neurons. 

The parameter space, i.e., (T, $) where $ is the 
parameter that controls now the noise intensity, 
also depicts the P and F phases, even for the case of 
sequential updating 1251 . The corresponding tran- 
sition results of second order only for $ > —4/3, 
which includes the Hopfield classical limit <I> = — 1 
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Figure 4: Time evolution of every overlap during a 
Monte Carlo simulation with 400 neurons, M — 5 com- 
pletely uncorrelated patterns, $ = 0.05, and T = 0.1. 
The system in this case was perturbed by the stimulus 
— (55"^, 5 — 0.3, with the pattern index v changed at ran- 
dom every 20000 MC steps. The same experiment with 
"I> = —1; i.e., the Hopfield case, produces a constant 
signal instead. 



(lack of depression). Otherwise, the transition is 
of first-order — with excellent recognition memory 
due to the sharp behavior just below the transition 
temperature. More specifically, there is a tricritical 
point is at (Te,$c) = (1,-4/3). 

Also interesting are the consequences of the 
activity-dependent processes — which mimic short- 
term synaptic depression in the model — on the re- 
trieval dynamics under external stimulation. That 
is, one may check the resulting sensitivity of the 
network to external inputs. A high degree of sensi- 
bility will facilitate the response to changing stim- 
uli. In fact, this is an important feature of neuro- 
biological systems which continuously adapt, and 
may thus quickly respond to varying stimuli from 
the environment. A simple external input may be 
simulated by adding to each local field a driving 
term -6^i, Vi, with < (5 < 1 [T^]. For a sin- 
gle pattern, M = 1, this tries to move the net- 
work activity from the attractor, ^, to the "antipat- 
tern", — f. The resulting behavior is illustrated in 
figure 13 and in the paper j25| . The graphs in this 
figure, and similar evidences can be obtained for 
other parameter values, clearly demonstrate that 




Figure 5: The top graph shows, for M = 1, the abso- 
lute value of the overlap as a function of the (neuron) 
temperature for "1> — 0.05. This illustrates, in a MC sim- 
ulation with 10^ neurons, irregular behavior and a tran- 
sition at Tc = 1 from the F phase to the P phase as the 
temperature is increased. The bottom graph shows the 
corresponding Lyapunov exponent, as obtained from an 
accurate mean-field description. This illustrates even- 
tual eventual chaotic behavior, i.e., A > 0. 



presynaptic noise enhances the network sensitiv- 
ity to a simple external stimulus. Furthermore, 
the jumping phenomena is robust with respect to 
the type of pattern stored. This is illustrated by 
comparison of the situation in figure 0] for uncorre- 
lated patterns, i.e., patterns with mutual overlaps 
m"^'' = 1/NY^-^^S.t « 0, and the one in Ref.|^ 
for correlated patterns, namely, m'^''' = 1/3 for any 
two of them. On the contrary, there is no visible 
structure in the signal response in the absence of 
fast noise, e.g., in the Hopfield case as far as (5 ^ 1. 
As a matter of fact, the depth of the basins of at- 
traction are large enough in the Hopfield model, to 
prevent any jumping phenomena for small S, except 
when approaching a critical point {Tc = 1), where 
fluctuations diverge. That is, adding fast noise in 
general seems to destabilize the fixed point for the 
interesting case of small 6 far from criticality. 

Summing up, the "two-temperatures model" I21j 
may naturally be adapted to incorporate certain 
neurobiological observations to obtain an interest- 



ing algorithm for memory recognition, including 
family discrimination. Using the language of neu- 
ral networks, the model has two parameters, say, 
"temperatures" which control the stochastic dy- 
namics of neurons and synaptic weights, respec- 
tively. We adapted this to study the consequences 
of fast synaptic noise on the attractor of a system 
with a finite number of stored patterns. Among 
the cases already considered, the most intriguing 
behavior ensues when the noise depends on the to- 
tal presynaptic current arriving to the postsynaptic 
neuron. This case has been studied both numer- 
ically and analytically by using a mean-field hy- 
pothesis. The numerical work consists of a series 
of Monte Carlo simulations using both sequential 
Glauber, spin-flip dynamics and cellular automa- 
ton or Little (parallel updating) dynamics. Most 
available results concern the limit a = M/N -^ 
but preliminary Monte Carlo simulations indicate 
that our results also hold for a macroscopic number 
of stored patterns, a ^ 0. 

A principal conclusion is that fast presynaptic 
noise may induce a noncquilibrium condition which 
results in an important intensification of the net- 
work sensitivity to external stimulation. One ex- 
plicitly sees that the noise may turn unstable the 
attractor or fixed point solution of the retrieval pro- 
cess, and the system then seeks for another attrac- 
tor. In particular, one observes jumping from the 
stored pattern to the corresponding antipattern for 
M — 1, and jumping between patterns for a larger 
number of stored patterns. This behavior is most 
interesting because it improves the network ability 
to detect changing stimuli from the environment. 
We observe the jumping to be very sensitive to the 
forcing stimulus, but rather independent of the net- 
work initial state or the thermal noise. We also ob- 
serve that the jumping may become chaotic when 
the system is implemented as a cellular automaton 
with parallel updating. This is illustrated in figure 

El 

Finally, one may argue that, besides recognition, 
the processes of class identification and categoriza- 
tion in nature might follow our model strategy. 
That is, different attractors may correspond to dif- 
ferent objects, and a dynamics conveniently per- 
turbed by fast noise may keep visiting the attrac- 
tors belonging to a class which is characterized by a 
certain degree of correlation between its elements. 
In fact, as described above, it was recently reported 



that a similar mechanism seems at the basis of sev- 
eral natural phenomena. 
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